Image pickup apparatus with white balance extraction

ABSTRACT

An image pickup apparatus which can improve convenience of white sheet data used for correcting the color of a picked-up image. Each image pickup data is converted into a file by an image filing unit and recorded by a medium-recording unit. Average value data of each color component data for part of the picked-up data is stored in an area different from the image pickup data recording area in each file.

This application is a continuation of U.S. Application Ser. No.12/118,434, filed May 9, 2008 (pending), which was a continuation ofU.S. application Ser. No. 11/495,647, filed Jul. 31, 2006 (now U.S. Pat.No. 7,385,633 issued on Jun. 10, 2008), which was a division of U.S.application Ser. No. 09/978,579, filed Oct. 18, 2001 (now U.S. Pat. No.7,133,071 issued on Nov. 7, 2006), which are incorporated by referenceherein in their entirety, as if fully set forth herein, and claim thebenefit of priority under 35 U.S.C. §119, based on Japanese PriorityApplication No. JP 2000-319718, filed Oct. 19, 2000, which isincorporated by reference herein in its entirety, as if fully set forthherein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image pickup apparatus for adjustingtone of image data picked up by an image pickup element having aplurality of color filters.

2. Related Background Art

To output image data picked up by an image pickup element having aplurality of color filters, the color temperature of a light source inthe picked-up image, and the picked-up image data undergoes whitebalance adjustment, thereby performing faithful color reproduction ofthe picked-up image.

The white balance adjustment methods include a method of determining thecolor temperature of a light source using a calorimetric element and amethod of searching for an achromatic portion in the picked-up image anddetermining the color temperature of the achromatic portion. Thesemethods, however, cannot accurately adjust the white balance due todetection errors of colors of the light sources.

As one of the most accurate white balance adjustment methods, a whitebalance adjustment method is available in which an achromatic object ispicked up (this operation is called white sheet image pickup), a controlvalue is obtained so as to equalize the magnitudes of color componentsof the picked-up image, and white balance adjustment is performed forthe subsequent image pickup using the control value. This white balanceadjustment method will be referred to as a manual white balance (MWB)hereinafter.

A conventional MWB control method using white sheet image pickup isshown in FIG. 11. FIG. 11 shows the conventional white balanceadjustment method of partially extracting data (to be referred to whitesheet data hereinafter) from an achromatic image (to be referred to as awhite sheet image hereinafter) picked up by an image pickup elementhaving a plurality of color filters, recording the extracted white sheetdata on a recording medium, reading the white sheet data from the mediumin the subsequent image pickup to calculate a white balance controlvalue, and adjusting the white balance using the calculated controlvalue.

The plurality of color filters of the image pickup element are arrayseach made up of primary color filters of R, G1, G2, and B, as shown inFIG. 5. Image pickup data input from an image pickup data input terminal1001 is processed by a main image processing unit 1002, encodingprocessing unit 1003, and image filing unit 1004. The processed data isrecorded on a recording medium as image data by a medium-recording unit1005.

Independently of this normal image recording process, part of the imagepickup data is extracted by an image data extraction unit as MWB whitesheet data. The extracted MWB white sheet data is recorded in the MWBdata area of the recording medium.

When the white balance mode is set in the MWB in the subsequent imagepickup, a medium-reproduction unit 1007 in the image pickup apparatusreproduces the white data recorded on the medium. A white balancecontrol value read-out unit 1008 reads out the white data. A whitebalance control value operating unit 1009 calculates the R, G1, G2, andB white balance control values so as to equalize the magnitudes of theaverage values of the R, G1, G2, and B color signals of the white data.

The white balance control values obtained by the white balance controlvalue operating unit 1009 are set for white balance adjustment of theimage pickup apparatus by a white balance control value setting unit1010. Using the set control values, a white balance adjustment unit 1014adjusts the white balance of the image pickup data input from an imagepickup data input terminal 1013 in the subsequent image pickup. A colorprocessing unit 1015 performs final color adjustment such as colorcorrection processing using a color matrix. An encoding processing unit1016 encodes the color-adjusted data into a recording format. The dataoutput from the encoding processing unit 1016 is converted into arecording file by an image filing unit 1017. The resultant file isrecorded on a recording medium 1018 as a pickup image file.

As shown in FIG. 12, in the MWB mode, a white balance control valueoperating unit 1102 directly calculates white balance control valuesfrom the white sheet data picked up by an image pickup apparatus (inputfrom an image pickup data input terminal 1101). The calculated controlvalues are set in the image pickup apparatus by a white balance controlvalue setting unit 1103 and used as the control values in subsequentimage pickup for a white balance adjustment unit 1114.

By using the set control values, the white balance adjustment unit 1114adjusts the white balance of the image pickup data input from an imagepickup data input terminal 1113 in the subsequent image pickup. A colorprocessing unit 1115 performs final color adjustment such as colormatrix conversion and color correction processing. An output from thecolor processing unit 1115 is encoded into a recording format by anencoding processing unit 1116. An output from the encoding processingunit 1116 is converted into a recording file by an image filing unit1117. The resultant file is recorded on a recording medium 1118 as apickup image file.

In the prior arts described above, however, white sheet data is writtenin a specific area called the MWB data area of a recording medium. Therecording medium is inserted into an image pickup apparatus to read outthe white data in the MWB mode. It is difficult to store a plurality oftypes of MWB white sheet data on one recording medium, manage them, andread out desired white data. The white data cannot be conveniently used.

The method of directly calculating the white balance control values uponpicking up image pickup data and setting them in the image pickupapparatus cannot set an another MWB mode when the white data is preparedonce and continuously used, resulting in inconvenience. The white datacannot be conveniently used either.

When white sheet data picked up by a given image pickup apparatus isused as white data for another image pickup apparatus in the abovesystem, accurate white balance adjustment cannot be performed due to theinfluence of variations in spectral characteristics of the image pickupelements. The white sheet data cannot be conveniently used either.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the conventionalproblems described above, and has its object to provide an image pickupapparatus in which white sheet data used for color correction of apicked-up image can be conveniently used, a signal processing method forthe image pickup apparatus, and a recording medium having a module forexecuting the signal processing of the signal processing method.

In order to achieve at least one of the above objects, according to anaspect of the present invention, there is provided an image pickupapparatus comprising: a memory for storing as white data first imagedata obtained from an image pickup element; and a control unit forconverting second image data from the image pickup element into a fileand storing the first image data stored the memory into an areadifferent from that of the second image data in the image data file.

The above and other objects, features, and advantages of the presentinvention will be apparent from the following detailed description inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the main part of an image pickupapparatus according to the first embodiment;

FIG. 2 is a block diagram showing the main part of an image pickupapparatus according to the second embodiment;

FIG. 3 is a block diagram showing the main part of an image pickupapparatus according to the third embodiment;

FIG. 4 is a block diagram showing the main part of an image pickupapparatus according to the fourth embodiment;

FIG. 5 is a view showing the structure of primary color filter arrays ofimage pickup elements;

FIG. 6 is a view showing the structure of complementary color filterarrays of image pickup elements;

FIG. 7 is a view showing the arrangement of a user interface accordingto the first and second embodiments;

FIG. 8 is a view showing the arrangement of a user interface accordingto the third embodiment;

FIG. 9 is a view showing image pickup data to be extracted according tothe first to fourth embodiments;

FIG. 10 is a view for explaining the order of operation for adjustingvariations in image pickup elements according to the fourth embodiment;

FIG. 11 is a block diagram showing a prior art; and

FIG. 12 is a block diagram showing another prior art.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 is a block diagram showing the main part of an image pickupapparatus according to the first embodiment. This embodiment willexemplify a method of writing white image data, which is obtained bypicking up the image of a white sheet in order to use it as whitebalance data, into an attached data area of each image file when eachpickup image is recorded on a recording medium.

Referring to FIG. 1, an image pickup data input terminal 101 receivesimage pickup data obtained by an image pickup element having four, R,G1, G2, and B, color filters and converted into digital data.

This image pickup data is processed in a main image processing unit 102for various image processing operations for recording with high imagequality as in a digital image recording apparatus (image pickupapparatus) such as a general digital camera. The processed image pickupdata is then encoded by an encoding processing unit 103 in accordancewith a JPEG recording format.

On the other hand, to obtain MWB data, the image of white sheet ispicked up, and the resultant data is held as image pickup data. Part ofthe MWB image pickup data, e.g., pixel data of 64 pixels in the centralportion of the two-dimensional image space of one picture is extractedby a white sheet data extraction unit 104, as shown in FIG. 9. Anaveraging processing unit 105 obtains an average value of 16-pixel dataof each color component signal of R, G1, G2, or B.

An image filing unit 106 forms an image file into which the averagevalue of the pixel data of each color component obtained as describedabove is converted together with image pickup data to be actuallyreproduced. The average value is allocated to a predetermined MWB whitesheet data area attached to an image file. A medium-recording unit 107records the average value together with the original encoded image dataas one image file on a detachable recording medium such as a flashmemory.

In this embodiment, since the white sheet data is obtained by extractingan image of 64 pixels in the central portion of the picture, any colorimage except white such as a white sheet can be picked up around the64-pixel area. When only white image data is used, the entire picturemust be set white by zooming or the like.

A technique for setting in the picked-up image white data for adjustingthe white balance in the MWB mode will be described below. A medium onwhich an image file is recorded by the above recording scheme is loadedin the image pickup apparatus, and image reproduction is done using auser interface shown in FIG. 7.

A PLAY button 703 shown in FIG. 7 is depressed to reproduce and displayimage data (image pickup data) serving as an actual reproduction targetof the image file on a reproduction monitor 701.

The operator sequentially switches a plurality of files with fileselection switches 704 and 705 while viewing the pickup image displayedon the reproduction monitor 701 and a white image 709 superposed on thepickup image. The operator then selects one image file obtained bypicking up the image of a white sheet for white balance data.

When one image file is selected, a white sheet data reproduction unit110 reproduces the white data on the monitor 701 on which the imageserving as the reproduction target is being reproduced.

File selection is done as follows. While a file in which white data usedas white balance data is written is being displayed on the monitor, aSELECT switch 708 is depressed to cause a white data image selectionunit 109 to select a desired one of a plurality of files.

As indicated by the reproduction monitor 701 in FIG. 7, if the imagepickup date and place are written in a location except a white datarecording area 702 when intentionally picking up an image used for whitebalance data, operation for the white data can be further facilitated.

Reproduction processing of an image in which white image data is storedwill be described below.

In accordance with an image file selection signal indicating that thefile is selected, the white sheet data image selection unit 109 extractsthe white sheet data of an image file being reproduced. A white balancecontrol value operating unit 111 calculates a white balance controlvalue on the basis of the white data, as will be described below.

White balance calculations are done as follows. The ratio of averagevalues of R, G1, G2, and B is calculated, and the reciprocal of theratio of R and B using the G1 and G2 average values as the median iscalculated as a white balance control value WBR, WBG1, WBG2 and WBB,wherein WBR=(G1+G2)/(2×R), WBG1=(G1+G2)/(2×G1), WBG2=(G1+G2)/(2×G2) andWBB=(G1+G2)/(2×B).

A white balance control value setting unit 112 sets the resultant whitebalance control value as the MWB control value in the image pickupapparatus.

When image pickup operation is performed using the MWB control value setas described above, a white balance adjustment unit 114 multiplies theset MWB control value with each color component, input from an imagepickup data input terminal 113, of an image signal output from the imagepickup element and converted into digital data, thereby adjusting thewhite balance.

The white balance-adjusted signal undergoes color processing in a colorprocessing unit 115 and encoding processing in an encoding processingunit 116. An image filing unit 117 converts the encoded image signalinto an image file together with the white sheet data extracted from theimage pickup data. A medium-recording unit 118 records the image file ona recording medium.

As described above, there is provided a function of setting in the imagepickup apparatus the MWB control value obtained from the white sheetdata recorded together with the image pickup data in the image file.White data recorded in a plurality of scenes can easily be repeatedlyused. Image pickup operation for setting an MWB control value upon achange in scene need not be done, thereby facilitating the use of MWBadjustment. Note that in this embodiment, it is described fordescriptive simplicity that the image is reproduced on the monitor 701in the image pickup apparatus, but an image file may be reproducedusing, e.g., a personal computer.

As described above, according to the first embodiment, a plurality ofwhite sheet data can be managed using one medium and accessed easily,thereby improving convenience in use of MWB white sheet data.

FIG. 2 is a block diagram showing the main part of an image pickupapparatus according to the second embodiment. This embodiment willexplain a method of recording part of image pickup data as coloradjustment data (to be referred to as color balance data hereinafter) ofa picked-up image in a data area of each image file when recording eachpickup image on a recording medium.

Referring to FIG. 2, an image pickup data input terminal 201 receivesimage pickup data obtained by an image pickup element having four, R,G1, G2, and B, color filters and converted into digital data.

This image pickup data is processed in a main image processing unit 202with various image processing operations for recording with high imagequality as in a digital image recording apparatus such as a generaldigital camera. The processed image pickup data is then encoded by anencoding processing unit 203 in accordance with a JPEG recording format.

On the other hand, part of the image pickup data, e.g., pixel data of 64pixels in the central portion of the two-dimensional image space of onepicture is extracted by a color balance data extraction unit 204, asshown in FIG. 9. An averaging processing unit 205 generates an averagevalue of 16-pixel data of each color component signal of R, G1, G2, orB.

An image filing unit 206 forms an image file into which the resultantaverage value is converted together with encoded image data. The averagevalue is allocated to a predetermined color balance data area attachedto an image file. A medium-recording unit 207 records the average valuetogether with the original encoded image data as one image file on arecording medium.

The above operation is always done in all image pickup operationsincluding normal image pickup. Color balance image pickup is done as oneof the general image pickup operations. Since color balance data isobtained by extracting an image of 64 pixels in the central portion ofthe picture, any color image except color balance data may be picked uparound the 64-pixel area.

A technique for setting in the image pickup apparatus a color balancecontrol value used for color balance adjustment after white balanceadjustment will be described below. A medium on which an image file isrecorded by the above recording scheme is loaded in the image pickupapparatus, and image file reproduction is done using a user interfaceshown in FIG. 7.

A PLAY button 703 shown in FIG. 7 is depressed to reproduce and displayimage data serving as an actual reproduction target of the image file,on a reproduction monitor 701.

The operator sequentially switches a plurality of files with fileselection switches 704 and 705 while viewing the pickup image displayedon the reproduction motor 701 and an image 709 superposed on the pickupimage. The operator then selects one image file to be used for colorbalance adjustment among a plurality of image files.

When one image file is reproduced and monitored, a color balance datareproduction unit 210 reads out and reproduces the color balance datafrom the attached color balance data area.

File selection is done as follows. While a file in which color balancedata used for color balance adjustment is written is being displayed onthe monitor, a SELECT switch 708 is depressed to cause a color balancedata image selection unit 209 to select a desired file.

As indicated by the monitor 701 in FIG. 7, when the image pickup dateand place are written in a location except a color balance datarecording area 702 in intentionally picking up an image used for colorbalance adjustment, operation for the color balance data can be furtherfacilitated.

In accordance with an image file selection signal indicating that a fileis selected, the color balance data image selection unit 209 extractsthe color balance data of an image file being reproduced. A colorbalance control value operating unit 211 calculates a color balancecontrol value.

Color balance calculations are done as follows. The ratio of averagevalues of R, G1, G2, and B is calculated, and the reciprocal of theratio of R and B using the G1 and G2 average values as the median iscalculated as a color balance control value.

The color balance adjustment control value can be finely adjusted(changed) in the following manner.

For example, adjustment in the R-B direction is performed using theswitches 704 and 705 shown in FIG. 7. Every time the switch 704 isdepressed, a color balance adjustment coefficient unit 219 outputs an Rsum signal. A color balance control value operating unit 211 adds aconstant to the R control value and subtracts a constant from the Bcontrol value. Every time the switch 705 is depressed, the color balanceadjustment coefficient unit 219 outputs a B sum signal. The colorbalance control value operating unit 211 adds a constant to the Bcontrol value and subtracts a constant from the R control value.

Adjustment in the magenta-green direction is performed using switches706 and 707 shown in FIG. 7. Every time the switch 706 is depressed, thecolor balance adjustment coefficient unit 219 outputs an R+B sum signal.The color balance control value operating unit 211 adds a constant tothe R and B control values and subtracts a constant from the G1 and G2control values. Every time the switch 705 is depressed, the colorbalance adjustment coefficient unit 219 outputs a Gr sum signal. Thecolor balance control value operating unit 211 adds a constant to the G1and G2 control values and subtracts a constant from the R and B controlvalues.

The R, G, and B signals for adjusting the output signals of a monitor(not shown) can be similarly controlled by the signals from the colorbalance adjustment coefficient unit 219 to allow the operator to confirmthe fine adjustment on the monitor.

The color balance control values obtained as described above are set inthe image pickup apparatus as the color balance adjustment controlvalues by a color balance control value setting unit 212.

As described above, when image pickup operation is performed using thecolor balance adjustment control values thus set, an auto white balanceadjustment unit 219 adjusts the white balance of each color component ofa digital image signal input from an image pickup data input terminal213 via an image pickup element. A color balance adjustment unit 214multiplies the set color balance control value with each colorcomponent, thereby adjusting the color balance.

For example, when an image has a uniform red portion in its center, areddish image is output since this reddish tincture is reflected on theoutput image.

The color balance-adjusted signal undergoes color processing in a colorprocessing unit 215 and encoding processing in an encoding processingunit 216. An image filing unit 217 converts the encoded image signalinto an image file together with the color balance data extracted fromthe image pickup data. A medium-recording unit 218 records the imagefile on a recording medium.

As described above, there is provided a function of setting in the imagepickup apparatus the color balance adjustment control value obtainedfrom the color balance data recorded together with the image pickup datain the image file. Color balance data recorded in a plurality of scenescan be easily selected and repeatedly used. The user can freely expressan image by easily changing the color tincture of the pickup image.

As described above, according to the second embodiment, color balancedata recorded in a plurality of scenes can be easily selected andrepeatedly used. The user can freely change the color balance of theauto white balance-adjusted pickup image. The user can freely express animage by easily changing the color tincture of the pickup image.

FIG. 3 is a block diagram showing the main part of an image pickupapparatus according to the third embodiment. This embodiment willexplain a method of recording part of image pickup data as white sheetdata or color balance data of a picked-up image in a data area of eachimage file when recording each pickup image on a recording medium, andusing these data for MWB adjustment and color balance adjustment.

Referring to FIG. 3, an image pickup data input terminal 301 receivesimage pickup data obtained by an image pickup element having four, R,G1, G2, and B, color filters and converted into digital data.

This image pickup data is processed in a main image processing unit 302for various image processing operations for recording with high imagequality as in a digital image recording apparatus such as a generaldigital camera. The processed image pickup data is then encoded by anencoding processing unit 303 in accordance with a JPEG recording format.

On the other hand, part of the image pickup data, e.g., pixel data of 64pixels in the central portion of the two-dimensional image space of onepicture is extracted by an image data extraction unit 304, as shown inFIG. 9. An averaging processing unit 305 generates an average value of16-pixel data of each color component signal of R, G1, G2, or B.

An image filing unit 306 forms an image file into which the resultantaverage value is converted together with encoded image data. The averagevalue is allocated to a predetermined area attached to an image file. Amedium-recording unit 307 records the average value together with theoriginal encoded image data as one image file on a recording medium.

The above operation is always done in all image pickup operationsincluding normal image pickup. Each of white sheet data image pickup andcolor balance image pickup is done as one of the general image pickupoperations. Since the white sheet data and color balance data areobtained by extracting an image of 64 pixels in the central portion ofthe picture, any image except white sheet data and color balance datamay be picked up around the 64-pixel area.

A technique for setting in the image pickup apparatus an MWB controlvalue used for MWB adjustment and a color balance control value used forcolor balance adjustment will be described below. A medium on which animage file is recorded by the above recording scheme is loaded in theimage pickup apparatus, and image file reproduction is done using a userinterface shown in FIG. 8.

A PLAY button 803 shown in FIG. 8 is depressed to reproduce and displayimage data serving as an actual reproduction target (image pickup data)of the image file on a reproduction monitor 801.

The operator sequentially switches a plurality of files with fileselection switches 804 and 805 while viewing the pickup image displayedon the reproduction motor 801 and a white image or color adjustmentimage 810 superposed on the pickup image. The operator then selects oneimage file to be used for white balance adjustment.

When one image file is reproduced and monitored, a selection datareproduction unit 310 reads out and reproduces the white sheet data fromthe attached data area.

File selection is done as follows. While a file in which white balancedata used for white balance adjustment is written is being displayed onthe monitor, a WHITE switch 808 is depressed to cause a CB/WB adjustmentimage selection unit (first selection means) 309 to select a desiredfile.

As indicated by the monitor 801 shown in FIG. 8, if the image pickupdate and place are written in a location except a white balance datarecording area 802 when intentionally picking up an image used for whitebalance adjustment, operation for the white sheet data can be furtherfacilitated. This also applies to color balance adjustment to bedescribed later.

The attached data of the image file being reproduced is selected by theCB/WB adjustment image selection unit 309 in accordance with the fileselection signal indicating that a file is selected. Upon depressing theWhite switch 808, a switch 321 is switched in accordance with a signalfrom a CB/WB selection unit 320 for switching between white balance dataand the color balance data. A white balance control value operating unit311 calculates a white balance control value using the selected data.

White balance calculations are done as follows. The ratio of averagevalues of R, G1, G2, and B is calculated, and the reciprocal of theratio of R and B using the G1 and G2 average values as the median iscalculated as a white balance control value.

The white balance adjustment control value can be set in the imagepickup apparatus as the MWB adjustment control value by a CB/WB controlvalue setting unit 313.

When the MWB control value is set in the image pickup apparatus, a colorbalance control value is calculated in the same manner as in the MWBcontrol value and set in the image pickup apparatus.

The PLAY button 803 shown in FIG. 8 is depressed to cause amedium-reproduction unit 308 to reproduce and display image pickup dataon a reproduction monitor 701.

The operator sequentially switches a plurality of files with fileselection switches 804 and 805 while viewing the pickup image displayedon the reproduction motor 801. The operator then selects one image filedesired to be used for color balance adjustment.

While the image file is reproduced and monitored, a selection datareproduction unit 310 reads out the color balance data from the attacheddata area and reproduces it on the monitor 701.

File selection is done as follows. While a file in which data desired tobe used for color balance adjustment is being displayed on the monitor,a COLOR switch 809 is depressed to cause the CB/WB adjustment imageselection unit 309 to select a desired file.

In accordance with the file selection signal indicating that a file isselected, the CB/WB adjustment image selection unit 309 selects theattached data of the image file being reproduced. Upon depressing theCOLOR switch 809, the switch 321 is switched in accordance with theWB/CB selection unit 320 for switching between the white balance dataand the color balance data. A color balance control value operating unit312 calculates a color balance control value using the selected data.

Color balance calculations are done as follows. The ratio of averagevalues of R, G1, G2, and B is calculated, and the reciprocal of theratio of R and B using the G1 and G2 average values as the median iscalculated as a color balance control value.

The color balance adjustment control value can be finely adjusted inaccordance with a signal from a color balance adjustment coefficientunit 322 as in the second embodiment.

The color balance control values thus obtained are set in the imagepickup apparatus by the CB/WB control value setting unit 313 as thecolor balance adjustment control values.

When image pickup operation is performed using the white balanceadjustment control value and color balance adjustment control value setas described above, a white balance adjustment unit 320 performs MWBadjustment for each color component of a digital image signal input froman image pickup data input terminal 314 via an image pickup element,thereby adjusting the white balance.

A color balance adjustment unit 315 then performs color balanceadjustment. The set color balance control value is multiplied with eachcolor component to perform color balance adjustment.

The color balance-adjusted signal undergoes color processing in a colorprocessing unit 316 and encoding processing in an encoding processingunit 317. An image filing unit 318 converts the encoded image signalinto an image file together with the white sheet data and color balancedata extracted from the image pickup data. A medium-recording unit 319records the image file on a recording medium.

As described above, there is provided a function of setting in the imagepickup apparatus the MWB adjustment control value and color balanceadjustment control value obtained from the color balance data recordedtogether with the image pickup data in the image file. Attached data ofimage files recorded in a plurality of scenes can be easily selected andrepeatedly used. The user can freely express an image by easily changingthe color tincture of the pickup image.

More specifically, MWB white sheet data is read out to perform whitebalance adjustment. The same white sheet data or another white sheetdata also can be read out to perform color balance adjustment. Thus,attached data of image files recorded in a plurality of scenes can beeasily selected and repeatedly used. The user can freely express animage by easily changing the color tincture of the pickup image.

FIG. 4 is a block diagram showing the main part of an image pickupapparatus according to the fourth embodiment. This embodiment willexplain a method of recording part of picked-up image as gain controldata for MWB and color balance adjustment in an attached data area ofeach image file, performing adjustment operation so as to use the gainadjustment data in any image pickup apparatus and perform MWB adjustmentand color balance adjustment in recording each pickup image on arecording medium.

A method of extracting some data from image pickup data and recordingthe extracted data as data for MWB or color balance adjustment in anattached area of each image file is the same as in the first embodiment,and a detailed description thereof will be omitted.

According to the characteristic feature of the fourth embodiment, imagepickup element fluctuation adjustment data recorded in an image pickupelement fluctuation adjustment data recording area (memory) oradjustment data unit 419 of this image file as data obtained by pickingup images of a white sheet under each of two different specific lightsources are recorded in a specific part of the attached area.

Image pickup element fluctuation adjustment data, i.e., image pickupdata 1203 and 1205 of two different light sources obtained by using areference image pickup apparatus shown in FIG. 10 and image pickup data1204 and 1206 of two different light sources unique to each image pickupapparatus for each color component of R, G1, G2, or B, are recorded bythe adjustment data unit 419 in the area attached to the image file.This embodiment uses two different light sources. However, adjustmentprecision is improved when the number of types of light sourcesincreases. Although the adjustment precision is not improved so much,the image pickup element fluctuation adjustment can be also attained byusing only one kind of light source.

These image pickup element fluctuation adjustment data which is writtenin the memory of each image pickup apparatus at the time of shipment,are read out by a readout means (not shown) in each image pickupoperation and then written in a single file together with the MWB whitesheet data and color balance data.

The average value of all or some image pickup data for each colorcomponent of R, G1, G2, or B, obtained when picking up the image of alight source, is used as such image pickup element fluctuationadjustment operation data.

As in the above three embodiments, when the attached data of a fileselected by a selection data reproduction unit 410 shown in FIG. 4 isread out, the fluctuation adjustment data are also read out andreproduced.

An adjustment operation unit 420 adjusts these fluctuation adjustmentdata in the following manner and supplies the resultant data as R, G1,G2, and B data for MWB or color balance adjustment to a gain controlvalue operating unit 4111. The gain control value operating unit 4111calculates gain control values for MWB and color balance adjustment.

Referring to FIG. 10, an image pickup apparatus 1201 records MWB andcolor balance adjustment data on a medium. An image pickup apparatus1202 reads out the MWB and color balance gain adjustment data recordedon the recording medium and performs gain adjustment.

Fluctuation adjustment between the image pickup elements in an imagepickup apparatus is two-dimensionally performed using ratios R/B and(R+B)/(G1+G2) of reference data Rrefa, G1 refa, G2 refa, Brefa of alight source a, reference data Rrefb, G1 refb, G2 refb, and Brefb of alight source b, and data Ra, G1 a, G2 a, and Ba of the light source a,and data RbS, G1 bS, G2 bS and BbS of the light source b which areunique to the image pickup apparatus.

The method of fluctuation adjustment calculation between the imagepickup elements in an image pickup apparatus can be expressed by adifference such as an (R−B) value or (R+B)−(G1+G2) value. Anymathematical expression can be used if it can be expressed so as toabsorb fluctuations between the image pickup elements in the imagepickup apparatus.

First of all, the relationship between the reference data RrefaS, G1refaS, G2 refaS, BrefaS, RrefbS, G1 refbS, G2 refbS, and BrefbS (1203)read out from a given recording medium and adjustment data RaS, G1 aS,G2 aS, BaS, RbS, G1 bS, G2 bS, and BbS (1204) unique to the image pickupapparatus and recorded on the given recording medium is adjusted usingMWB and color balance gain adjustment data R, G1, G2, and B read outfrom the given recording medium.

Gain values and offset values of the difference between the R/B ratiosand (R+B)/(G1+G2) ratios of the light sources a and b are obtained:

$\begin{matrix}{{GainRB} = {\{ {( {{RrefaS}/{BrefaS}} ) - ( {{RrefbS}/{BrefbS}} )} \}/\{ {( {{RaS}/{BaS}} ) - ( {{RbS}/{BbS}} )} \}}} & (1) \\{\mspace{20mu}{{OffsetRB} = {( {{RrefaS}/{BrefaS}} ) - {{GainRB}*( {{RaS}/{BaS}} )}}}} & (2)\end{matrix}$These are reflected on the gain adjustment data.(R′/B′)=GainRB*(R/B)+OffsetRB  (3)Similarly, the following relations are obtained for the (R+B)/(G1+G2)ratios:

$\begin{matrix}{{{GainRBG}\; 1G\; 2} = {\{ {{( {{RrefaS} + {BredfaS}} )/( {{GlrefaS} + {G\; 2{refaS}}} )} - {( {{RrefbS} + {BrefbS}} )/( {{GlrefbS} + {G\; 2{refbS}}} )}} \}/\{ {{( {{RaS} + {BaS}} )/( {{GlaS} + {G\; 2{aS}}} )} - {( {{RbS} + {BbS}} )/( {{GlbS} + {G\; 2{bS}}} )}} \}}} & (4) \\{{{OffsetRBG}\; 1G\; 2} = {\{ {( {{RrefaS} + {BrefaS}} )/( {{GlrefaS} + {G\; 2{refaS}}} )} \} - {{GainRBG}\; 1G\; 2*\{ {( {{RaS} + {BaS}} )/( {{GlaS} + {G\; 2{aS}}} )} \}}}} & (5) \\{{( {R^{\prime} + B^{\prime}} )/( {{G\; 1^{\prime}} + {G\; 2^{\prime}}} )} = {{{GainRBG}\; 1G\; 2*\{ {( {R + B} )/( {{G\; 1} + {G\; 2}} )} \}} + {{OffsetRBG}\; 1G\; 2}}} & (6)\end{matrix}$

These equations yield data R′, G1′, G2′ and B′ converted into referencedata for the image pickup apparatus which recorded the data on themedium. The resultant data is given by adjustment operation (1) 1207shown in FIG. 10.

Now assuming that G1′ and G2′ are fixed values respectively such asG1′=G1 refbS and G2′=G2 refbS, R′ and B′ can be calculated from theequations (3) and (6) which calculate R′/B′ and (R′+B′)/(G1′+G2′)respectively. The thus-obtained R′, G1′, G2′ and B′ are written by theimage filing unit 406 into a file same as the image file as referencewhite sheet data for MWB. A procedure 1208 is a processing method oftransferring the white sheet data for MWB to an other image pickupapparatus D 1202 through the image file thus-recorded on the recordingmedium.

Next, a method of setting the white sheet data for MWB to the imagepickup apparatus D 1202 will be explained. Since the MWB white sheetdata R′, G1′, G2′ and B′ are read out from the recording medium asreference white sheet data for MWB, an operation 2 which adjusts datawhose characteristic is suitable for the image pickup apparatus D 1202is performed using reference data 1205 and specific data 1206 of lightsources a and b of the image pickup apparatus D 1202. The adjustmentoperation is performed to calculate gain value and offset value ofdifference in a ratio R/B and a ratio (R+B)/(G1+G2) between the lightsource a and the light source b, whereinGainRB′={(RaD/BaD)−(RbD/BbD)}/{(RrefaD/BrefaD)−(RrefbD/BrefbD)}  (7)OffsetRB′=(RaS/BaS)−GainRB′*(RrefaS/BrefaS)  (8)

These results are reflected on gain adjustment data, that is,R″/B″=GainRB′*(R′/B′)+OffsetRB′  (9)

In the same manner, a calculation as to the ratio (R+B)/(G1+G2) isperformed as follows:

$\begin{matrix}{{{GainRBG}\; 1\; G\; 2^{\prime}} = {\{ {{( {{RaD} + {BaD}} )/( {{G\; 1{aD}} + {G\; 2{aD}}} )} - {( {{RbD} + {BbD}} )/( {{G\; 1{bD}} + {G\; 2{bD}}} )}} \}/\{ {{( {{RrefaD} + {BrefaD}} )/( {{G\; 1{refaD}} + {G\; 2{refaD}}} )} - {( {{RrefbD} + {BrefbD}} )/( {{G\; 1{refbD}} + {G\; 2{refbD}}} )}} \}}} & (10) \\{{{OffsetRBG}\; 1\; G\; 2^{\prime}} = {\{ {( {{RaS} + {BaS}} )/( {{G\; 1\;{aS}} + {G\; 2\;{aS}}} )} \} - {{GainRB}^{\prime}*\{ {( {{RrefaS} + {BrefaS}} )/( {{G\; 1\;{refaS}} + {G\; 2\;{refaS}}} )} \}}}} & (11) \\{{( {R^{''} + B^{''}} )/( \;{{G\; 1^{''}} + {G\; 2^{''}}} )} = {{{GainRBG}\; 1\; G\; 2^{\prime}*\{ {( {R^{\prime} + B^{\prime}} )/( {{G\; 1} + {G\; 2^{\prime}}} )} \}} + {{OffsetRBG}\; 1\; G\; 2^{\prime}}}} & (12)\end{matrix}$

Now assuming that G1 and G2′ are fixed values respectively such asG1′=G1 bD and G2′=G2 bD, R″ and B″ can be calculated from the equations(3) and (6) which calculate R″/B″ and (R″+B″)/(G1″+G2″) respectively.These R″, G1″, G2″ and B″ are set as the white sheet data for MWB.

The R″, G1″, G2″, and B″ values thus obtained represent values whichabsorb fluctuations between the image pickup elements in the imagepickup apparatuses 1201 and 1202 upon converting the data of the imagepickup apparatus 1201 (medium-providing-side image pickup apparatus S)into data of the image pickup apparatus 1202 (medium-use-side imagepickup apparatus D).

The gain adjustment control value operation unit 411 calculates theratio of R″, G1″, G2″, and B″ from the values obtained by the arithmeticoperations (1) and (2) and obtains the MWB and color balance adjustmentcontrol values WBR, WBG1, WBG2 and WBB, wherein WBR=(G1″+G2″)/(2×R″),WBG1=(G1″+G2″)/(2×G1″), WBG2=(G1″+G2″)/(2×G2″) and WBB=(G1″+G2″)/(2×B″).Then, the gain control value setting unit (setting means) 412 performssettings.

Using the gain adjustment control values thus set, the gain adjustmentunit 414 performs MWB or color balance adjustment in the subsequentimage pickup. The data are converted into an image file as in the aboveembodiments. The resultant data is recorded on a recording medium by amedium-recording unit 418.

By the above method, data written in the specific area attached to theimage file used for MWB and color balance adjustment is adjusted forfluctuations using the image pickup element fluctuation adjustmentoperation data of the image pickup apparatus, written in the samespecific area. Thus, as long as identical light sources are used, thesame R, G, G1, and B values are obtained even when different imagepickup apparatus are used. Therefore, data fluctuations between theimage pickup apparatuses can be absorbed.

As described above, even if MWB or color balance control values areobtained using the white sheet data or color balance adjustment data ofimage files picked up using different image pickup apparatuses, optimal(accurate) MWB or color balance adjustment operation can be done.

According to the fourth embodiment, as described above, white balanceadjustment or color balance adjustment using white sheet data can beaccurately done in an image pickup apparatus different from that bywhich the white sheet data are obtained.

In the first to fourth embodiments, the colors of the color filters ofthe image pickup element are R, G1, G2, and B. However, the presentinvention is also applicable to complementary color filters of Mg(magenta), Gr (green), Cy (cyan), and Ye (yellow) components, as shownin FIG. 6.

The object of the present invention is realized even by supplying astorage medium storing software program codes for realizing thefunctions of the above-described embodiments to a system or anapparatus, and causing the computer (or a CPU or an MPU) of the systemor the apparatus to read out and execute the program codes stored in thestorage medium.

In this case, the program codes read out from the storage medium realizethe functions of the above-described embodiments by themselves, and thestorage medium storing the program codes therefore constitutes thepresent invention.

As a storage medium for supplying the program codes, a floppy disk, ahard disk, an optical disk, a magnetooptical disk, a CD-ROM, a CD-R, amagnetic tape, a nonvolatile memory card, a ROM, or the like can beused.

The functions of the above-described embodiments are realized not onlywhen the readout program codes are executed by the computer but alsowhen the OS (Operating System) running on the computer performs part orall of actual processing on the basis of the instructions of the programcodes.

The functions of the above-described embodiments are also realized whenthe program codes read out from the storage medium are written in thememory of a function expansion board inserted into the computer or afunction expansion unit connected to the computer, and the CPU of thefunction expansion board or function expansion unit performs part or allof actual processing on the basis of the instructions of the programcodes.

When the present invention is applied to the above storage medium,program codes corresponding to the flow chart described above are storedin the storage medium. Briefly speaking, modules indispensable to theimage pickup apparatus of each embodiment described above are stored inthe storage medium.

As has been described above, according to the present invention,convenience of the white sheet data used for color correction of apicked-up image can be improved.

Many widely different embodiments of the present invention may beconstructed without departing from the spirit and scope of the presentinvention. It should be understood that the present invention is notlimited to the specific embodiments described in the specification,except as defined in the appended claims.

1. An image pickup apparatus comprising: an extracting unit constructedto extract information for white balance calculation from first imagedata obtained from an image pickup element, wherein the extractedinformation for white balance calculation includes calculation valuescorresponding to each of colors from the first image data; a storingunit constructed to store a plurality of files in a memory, wherein theplurality of files include the first image data obtained from the imagepickup element and the extracted information for white balancecalculation including the calculation values corresponding to each ofcolors from the first image data; a display unit which displaysinformation of at least one of the plurality of files including thefirst image data and the extracted information for white balancecalculation, wherein the display information includes at least one ofthe first image data and attribute data of the first image data; aselection member constructed to accept selection of a file from theplurality of files including the first image data of which informationis displayed on the display unit; and a calculation unit constructed tocalculate a white balance setting parameter for second image data byusing the extracted information for white balance calculation of thefirst image data included in the file selected by the selection member,wherein the second image data is new image data which is picked up aftersaid selection member selects the file.
 2. The image pickup apparatusaccording to claim 1, wherein the extracted information for whitebalance calculation is specified after the image pickup operation forthe first image data is implemented in accordance with a predeterminedsetting.
 3. The image pickup apparatus according to claim 1, wherein theselection member is disposed on a surface of the image pickup apparatus.4. The image pickup apparatus according to Claim 1, wherein thecalculation values are average pixel values corresponding to each ofpixel colors from a center area of the first image data.
 5. The imagepickup apparatus according to Claim 1, further comprising an adjustingunit constructed to adjust a white balance of the second image dataobtained from the image pickup element, in accordance with thecalculation by the calculation unit.
 6. The image pickup apparatusaccording to claim 5, further comprises a control unit which controlsstorage of the second image data adjusted by the adjusting unit as afile different from the files of the first image data.
 7. A controlmethod of an image pickup apparatus including an image pickup elementand a display unit, comprising the steps of: extracting information forwhite balance calculation from first image data obtained from the imagepickup element, wherein the extracted information for white balancecalculation includes calculation values corresponding to each of colorsfrom the first image data; storing a plurality of files in a memory,wherein the plurality of files include the first image data obtainedfrom the image pickup element and the extracted information for whitebalance calculation including the calculation values corresponding toeach of colors from the first image data; displaying, on the displayunit, information of at least one of the plurality of files includingthe first image data and the extracted information for white balancecalculation, wherein the display information includes at least one ofthe first image data and attribute data of the first image data;accepting selection of a file from the plurality of files including thefirst image data of which information is displayed on the display unit;and calculating a white balance setting parameter for second image databy using the extracted information for white balance calculation of thefirst image data included in the file selected in the selectionaccepting step, wherein the second image data is new image data which ispicked up after the selection accepting step accepts the selection ofthe file.
 8. The method according to Claim 7, wherein the calculationvalues are average pixel values corresponding to each of pixel colorsfrom a center area of the first image data.
 9. The method according toClaim 7, further comprising adjusting a white balance of the secondimage data obtained from the image pickup element, in accordance withthe calculation in the calculation step.